Introduction to the Shared Task on Comparing Semantic Representations

Seven groups participated in the STEP 2008 shared task on comparing semantic representations as output by practical wide-coverage NLP systems. Each of this groups developed their own system for producing semantic representations for texts, each in their own semantic formalism. Each group was requested to provide a short sample text, producing a shared task set of seven texts, allowing participants to challenge each other. Following this, each group was asked to provide the raw system output for all texts, which are made available on http://www.sigsem. org. Two groups were extremely inspired by the shared task and also provided gold-standard semantic representations for the seven texts, together with evaluation measures. The STEP 2008 workshop itself will continue the discussion, focusing on the feasibility of a theory-neutral gold standard for deep semantic representations.